Image-based control of wire harness and connector manufacturing processes using artificial intelligence

ABSTRACT

An embodiment provides a method of operating wire harness manufacturing equipment, including: adding, using the manufacturing equipment, an element to a wire to form a combination of the element and the wire; capturing, using an imaging device, one or more of an upper image and a lower image of the combination; analyzing, using one or more processors operatively coupled to the imaging device, the one or more of the upper image and the lower image to detect a defect; and thereafter indicating that the defect has been detected. Other embodiments are described and claimed.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/994,740, filed Aug. 17, 2020, which claims priority to U.S.provisional patent application Ser. No. 62/888,531, filed Aug. 18, 2019,each having the same title, the contents of which are incorporated byreference in their entirety herein.

FIELD

The subject matter disclosed herein generally relates to image-basedquality control of wire harness and connector manufacturing equipment,hereafter “wire harness manufacturing equipment,” such as hand tools,manual crimp presses, automatic wire processing machines, and ultrasonicwelding machines.

BACKGROUND

Many industries, such as the automotive, construction, aerospace, andhome appliance industries, rely on wiring for power and datacommunication. The wiring is typically gathered together in a groupingreferred to as a wire harness. Each individual wire or cable istypically made of an insulating material that surrounds a conductivematerial, such as copper. A wire is often finished at the ends by theaddition of a terminal, splice or weld, e.g., resistance or ultrasonicwelds, to make the wire functionally useful for the given application.

The process of adding an element such as a terminal to a wire caninvolve mechanically crimping a terminal onto an end of the wire thathas a predetermined amount of conductive material exposed from theinsulating material. Various other additions may be made, e.g., splicingmultiple wires together via mechanically added splice terminal,ultrasonically welding multiple wires together to form a splice,addition of a seal to a wire, etc. Such wire processing may be performedmanually, e.g., using hand tools, a manual crimp press, orautomatically, e.g., with an automated wire processing machine, crimpingpress or welding machine, or semi-automatically.

In order to monitor the output of these processes, various monitoringtechniques have been developed. For example, a crimp force monitor (CFM)may be utilized to track the forces applied during crimping and outputan error indication if the monitored forces deviate from a “learned”reference indicating a potential defect. Visual inspections are oftenutilized in addition to crimp force monitoring in order to assist in thedetection of defects, e.g., defects that are difficult or impossible todetect using a CFM. For example, an operator may carry out a visualinspection manually to determine if the crimping process hassuccessfully added a terminal to the end of a wire without any strandsbeing pressed on top of the conductor crimp. In a computer-aided visionsystem, a camera system may image the wire and terminal combination inorder to detect known defects using a software process.

BRIEF SUMMARY

In summary, one embodiment provides a method of operating wire harnessmanufacturing equipment, comprising: adding, using the wire harnessmanufacturing equipment, an element to a wire to form a combination ofthe element and the wire; capturing, using an imaging device, one ormore of an upper image and a lower image of the combination; analyzing,using one or more processors operatively coupled to the imaging device,the one or more of the upper image and the lower image to detect adefect; and thereafter indicating that the defect has been detected.

Another embodiment provides a method, comprising: adding, using wireharness manufacturing equipment, an element to a wire to form acombination of the element and the wire; capturing, using an imagingdevice, one or more of an upper image and a lower image of thecombination; analyzing, using one or more processors operatively coupledto the imaging device, the one or more of the upper image and the lowerimage to detect if a defect is present; thereafter indicating that thedefect has not been detected; and incorporating at least the combinationof the element and the wire into a wire harness.

A further embodiment provides an executable program product thatincludes code that obtains, from an imaging device, one or more of anupper image and a lower image of a wire and element combination;analyzes, using one or more processors, the one or more of the upperimage and the lower image to detect a defect; and thereafter indicatesthat the defect has been detected.

A still further embodiment includes a method, comprising: obtaining,from an imaging device, one or more of an upper image and a lower imageof an element added to a wire to form a combination of the element andthe wire; analyzing, using one or more processors, the one or more ofthe upper image and the lower image to detect a defect; and thereafterindicating that the defect has been detected.

A yet further embodiment includes a product formed by one or more of theforegoing processes.

The foregoing is a summary and thus may contain simplifications,generalizations, and omissions of detail; consequently, those skilled inthe art will appreciate that the summary is illustrative only and is notintended to be in any way limiting.

For a better understanding of the claimed embodiments, reference is madeto the following description and the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an example automated crimping system according to anembodiment.

FIG. 2 illustrates an example method for image-based quality controlaccording to an embodiment.

FIG. 3 illustrates an example method for generating an artificial neuralnetwork according to an embodiment.

FIG. 4 illustrates an example computer system according to anembodiment.

FIG. 5 illustrates an example imaging device according to an embodiment.

FIG. 6 illustrates an example imaging device according to an embodiment.

FIG. 7A illustrates an example method of controlling a manual pressusing image-based defect detection according to an embodiment.

FIG. 7B illustrates an example method of controlling an automatic wireprocessing machine using image-based defect detection according to anembodiment.

FIG. 8 (A-B) illustrates an example of image-based defect detectionaccording to an embodiment.

FIG. 9 illustrates a process for manufacturing a product according to anembodiment.

FIG. 10 illustrates an example of a feature of interest embedded in arandom neural network training image.

DETAILED DESCRIPTION

It will be readily understood that the components of the embodiments, asgenerally described and illustrated in the figures herein, may bearranged and designed in a wide variety of different configurations inaddition to the described example embodiments. Thus, the following moredetailed description of the example embodiments, as represented in thefigures, is not intended to limit the scope of the claims but is merelyrepresentative of those embodiments.

Reference throughout this specification to “embodiment(s)” (or the like)means that a feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment,which may or may not be claimed. Thus, appearances of the phrases“according to embodiments” or “an embodiment” (or the like) in variousplaces throughout this specification are not necessarily all referringto the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. In thefollowing description, numerous specific details are provided to give athorough understanding of example embodiments. One skilled in therelevant art will recognize, however, that aspects can be practicedwithout one or more of the specific details, or with other methods,components, materials, etc. In other instances, well-known structures,materials, or operations are not shown or described in detail to avoidobfuscation.

Existing visual inspection techniques for quality control in wireharness manufacturing typically fall into two categories, i.e., manualinspection and computer-aided vision inspections. Each of theseconventional techniques is difficult to set up, implement, and can beunreliable in certain circumstances.

Having an operator visually inspect terminal additions, seal additions,welds or like additions, either with an unaided eye, with a microscope,or even using cross-section analyses, relies on the operator's skill,judgement and attention to detail. Further, even if performed correctly,operator visual inspection is labor intensive, expensive and typicallyonly about 80% effective (i.e., only about 80% of defects are detected).

Conventional computer-aided vision systems address some of theseconcerns; however, existing camera systems introduce other difficulties.Conventional automated camera systems, e.g., added into the process lineof an automated wire processing machine, require software customized,using a unique program, for each type of wire and element (e.g.,terminal) that is used. That is, these programs are created to analyze aparticular wire and element combination using a teach-in process. Theteach-in process typically includes creating a library of images, takenfor the particular wire and element combination. These images arecaptured using the specific machine, e.g., automated wire processingmachine, and camera set up in question. An operator then selects a bestor master image as a reference image. The master or reference image isused to organize specific inspection windows for defect detection. Thisteach-in process results in a specific program being built to analyzethe particular wire and element combination. If another wire and elementcombination is used, it requires loading of a different crimp inspectionprogram formed using a similar teach-in process.

Further, conventional vision systems rely on precise alignment of thewire and terminal combination such that the image captured issubstantially the same each time. That is, conventional vision systemscapture an image with precise timing and alignment, referenced to themaster or reference image of the teach-in process, for analysis. Thisprecise alignment is required for the system to reliably “find” theterminal in the camera's field of view and perform the programmedinspection tasks. Because conventional vision systems are sensitive tothe alignment of the wire and terminal combination, a teach-in processand adjustment of related inspection windows may be required at thestart of each new production run. That is, each new setup is unique interms of how the wire and terminal combination is presented to thecamera system, and therefore a teach-in process may be required for eachsetup and cannot be avoided. Conventional vision systems require aunique program to be created for each combination of terminal and wire.Any change to the system, e.g., changing wire type, terminal type, etc.,requires a new teach-in and program creation. Teach-in process andprogram creation is complex and requires a skilled person, normally nota machine operator.

Embodiments of the invention resolve the problems with conventionalvision inspection systems by introducing an artificial intelligenceprogram that significantly reduces the complexity of system set up andoperation without sacrificing quality. In an embodiment, an artificialintelligence program is utilized to identify the object of interest,e.g., a terminal element added to the end of an insulated wire. Thisobject identification is not sensitive to the alignment or orientationof the wire and terminal combination, i.e., so long as the object ofinterest is presented in the field of view of the camera(s), theinspection process may be successfully completed. Further, precisetiming is not essential, and embodiments can use video cameras to detectdefects using images of objects of interest, e.g., even if the object ofinterest is simply manually waived in the field of view of the camera(s)by an operator.

An embodiment uses the object identification to directly or indirectlyperform image-based defect detection. In an embodiment, the objectidentification can be used to choose subsequent image-based processingsteps. For example, the object identification may be used to set abounding box around the object and to thereafter place one or moreinspection windows in image area(s) of interest for additional imageprocessing, e.g., pixel-based RGB color analysis, hue light saturation(HLS) analysis, pattern matching, etc.

In an embodiment, an object's initial identification, e.g.,identification of a particular terminal type added to a wire, may beused to automatically select an appropriate image analysis program,e.g., selection of a specific neural network for that part. The initialidentification may use an artificial intelligence program or anothertype of identification, e.g., scanning of a code or manual entry.

In an embodiment, the object identification may be used directly fordefect detection. For example, the object identified in the image may bethe defect itself by training the neural network to identify the defectin question.

Further, embodiments described herein do not require a programselection, teach-in process, or program adjustments and therefore canbegin operating in a given installation without any customized set up.This makes the system much easier to install and use, particularly byinexperienced operators. Likewise, an embodiment permits easilyswitching between types of wires and elements that are to be analyzed.For example, an embodiment can detect an object type, e.g., crimpedterminal, splice, ultrasonic weld, resistance weld, etc., andautomatically implement the appropriate image processing steps foranalyzing the images and detecting the defects unique to that objecttype. Embodiments are therefore more flexible and generally applicableto various terminal, splice, and weld visual inspections that might beencountered in the production process.

The description now turns to the figures. The illustrated exampleembodiments will be best understood by reference to the figures. Thefollowing description is intended only by way of example and simplyillustrates certain selected example embodiments.

A system has been designed to recognize object characteristics throughcameras, e.g., video cameras, and a computer that processes the images,detects the desired characteristics, and makes a decision whether a partis accepted or rejected (contains a defect) according to the parametersprogrammed, e.g., trained into the neural network.

FIG. 1 illustrates an example of a process that may be controlledautomatically according to an embodiment. In the example of FIG. 1 , anautomated wire processing machine and related machinery are illustratedas system 100. This machinery includes components used to prepare a wirefor use in an automotive wire harness, for example. The system 100includes a cutting component 101, e.g., to cut an insulation coating ofthe wire 106. A robotic arm 105 may insert, remove and progress the wire106 from one component to the next, as generally illustrated by thedashed arrows in FIG. 1 . The wire 106 is progressed by robotic arm 105to a terminal crimping component 102, e.g., for adding a terminal 107 tothe wire 106 using a crimping process. The robotic arm 105 then movesthe wire 106 and terminal 107 combination to an imaging device 103 forimaging the top and bottom of the combination. If the image analysisprocess being applied indicates that there are no defects detected, therobotic arm 105 next takes the wire to a cutting component 104, whichcuts the wire 106 at an appropriate length for the given application.The system 100 can then repeat the process.

FIG. 2 illustrates an example process according to an embodiment. At 201the item to be visually inspected, e.g., a wire and an element such as aterminal, are moved, e.g., using robotic arm or manually, to an imagingarea, e.g., within the field of view of an imaging device with upper andlower cameras. One or more images are then captured of the combined wireand element at 202. This may include, for example, capturing an upperimage and a lower image of the wire and element combination atsubstantially the same time using one camera positioned above the wireand element combination and one camera positioned below the wire andelement combination.

At 203, the image(s) captured at 202 are analyzed for the presence of anobject of interest, e.g., the combination of the wire and a terminal. Inan embodiment, the object of interest is first identified and thereafteradditional image processing is applied to determine if a defect isdetected, as indicated at 204. In another embodiment, the object ofinterest is the defect itself, i.e., identification of the object candirectly lead to a defect detection decision, combining steps 203 and204.

Irrespective of the processing applied, an embodiment will indicate if adefect has been detected and choose different processing depending onthe result, as decision point 204 indicates. For example, if no defectis detected, an embodiment outputs an OK signal, which permits theprocess (e.g., robotic arm) to continue in its cycle or enables amachine to be utilized (e.g., allows a foot pedal of a manual pressingmachine to operate the press). If a defect has been detected, a NOT OKsignal is generated, which disables or inhibits the process (e.g.,robotic arm) from continuing in its cycle or disables a machine frombeing utilized (e.g., disables a foot pedal of a manual pressing machinefrom operating the press). In this way, image analysis is implemented tocontrol the processing of wire combinations (e.g., terminal additions,splices, ultrasonic welds, etc.) with respect to visual qualitydetection.

FIG. 3 provides an example method of preparing an artificial neuralnetwork to perform image analysis and object identification outlined insteps 203 and/or 204 of FIG. 2 . As illustrated in the process outlinedin FIG. 3 , an embodiment uses artificial neural networks to detectpatterns of the desired characteristics in the image obtained by thecamera(s). Such a process is much faster than comparing the imagesobtained by the camera using pixel analysis, as is commonly done inexisting visual inspection software. Further, the artificial neuralnetworks of an embodiment are not sensitive to many of the variablesthat perturb conventional systems, such as lighting or orientation oralignment of the object of interest. This in turn makes it possible touse smaller and low-resource hardware, e.g., such as the JETSON NANOfrom Nvidia Corporation and low-cost cameras, as further describedherein.

In an embodiment, an artificial neural network is created by anartificial intelligence script. Illustrated in FIG. 3 are example stepsfor generating an example artificial neural network. The steps of FIG. 3(or like steps) only have to be performed one time for the generation ofthe neural network. Thereafter, the system uses the generated neuralnetwork to process the images, without requiring a program selection,teach-in process, or program adjustments.

At 301, at least one feature image is obtained that includes the objector feature that is to be recognized. This image only contains an imageof the object or feature to be recognized, i.e., any unwanted featuresthat are not to be recognized should be removed from the image. Morethan one feature image may be used. For example, if the system is to betrained to recognize an object or feature in multiple colors, a featureimage of the feature in each such color should be obtained.

The feature image(s) obtained for the object or feature of interest areplaced in a particular location, e.g., folder or storage location, togenerate the artificial neural network. A program is then run togenerate the neural network. By way of example, the program downloads aset of random training images at 302, e.g., 10,000 or more public imagesfrom the Internet using Google Big Data Service, and stores them into astorage location or folder. Once downloaded, the training images can bereused for generating another neural network (e.g., to identify a secondfeature), i.e., there is no need to download new or more images, unlessmore images are desired to further train the neural networks. That is,with more training images, the learning will be more precise; althoughthe process will take more time to complete. In one example, the systemcan be programmed to automatically calculate the cost—benefit of suchadditional training and fix the number of training images at an optimalvalue for each learning phase, e.g., using a learning profit algorithm.

The program then combines (e.g., inserts or overlays) the feature imageinto each of the random training images at 303. An example overly isillustrated in FIG. 10 , which illustrates a feature image 1001 (crimpedwire terminal) overlaid on a training image 1002 (a dog picture). Anegative image includes a training image 1002 that lacks the featureimage 1001. In each image a distortion may be applied to the featureimage with the desired characteristic to modify its shape and positionslightly. This distortion is registered into a text file that saves theposition where the feature image has been placed inside the backgroundimage (random training image). This permits the software to know theground truth of where the feature image to be searched for is located inthe training image that has had the feature inserted at a randomposition. For example, a line of text is generated for each randomtraining image that is used as the background image for the featureimage. By way of specific example, at the end of the file name thecoordinates of the inserted feature image are specified, e.g. in aspreadsheet file where the line position is the image number and thetrailing number series are the coordinates of the feature image:

Line position: (file name-Positive-Coordinates)0015_0086_0033_0078_0045.jpg 1 86 33 78 45.

Thereafter, the software generates the images with the feature imagealready embedded in the background image (random training image), e.g.,as illustrated in the example of FIG. 10 , in another folder where thepositive images will be stored (images containing the object to be foundand the position where it is inside the image). Negative images are alsostored and used for training the artificial neural network. At the endof this procedure (steps 301-303), there are two folders or storagelocations, i.e., one with the background images alone (i.e., randomtraining images without the feature image inserted) and another with thesame images already with the feature image inserted. These folders areused to generate and train the artificial neural network, as illustratedat 304-305.

By way of specific example, using these files (N negative images and Npositive images), an embodiment starts neural network training forexample by using the OpenCV Version 3.4.1 library. An embodiment getsparameters, e.g., from OpenCV, to process the neural network, asindicated at 304. An embodiment creates N number of images from at least1 sample in order to train the neural network at 305. This eliminatesthe need to have N number of samples for this process, accelerating thetime for neural network generation. This further facilitates being ableto train another artificial neural network automatically without theneed for human intervention, since an embodiment takes the randomtraining images and generates positive and negative images, allowing thetraining to start the learning process automatically.

Once the learning process is complete, an embodiment automaticallychecks the decisions made by the neural network at 306 by comparing thedecisions made to the ground truth using the list of images and theposition where the feature image of the object or feature was placed.This permits an embodiment to recognize the effectiveness of the neuralnetwork by comparison to predetermined threshold(s) (e.g., threshold(s)for acceptable number of false positives, false negatives, or acombination thereof). If this trained artificial neural network meetswith the minimum threshold(s), as determined at 307, it is saved asindicated at 308, e.g., as an xml file in a storage location for use inthe production process (outlined in FIG. 2 ). Otherwise, the artificialneural network can be re-trained, e.g., by obtaining more random imagesand repeating the process, as indicated. A similar process can becompleted for each feature or object of interest, e.g., one trainedartificial neural network can be obtained for each combination of wireand terminal, or each splice, or each weld. The training may be perclass or type of feature or part, e.g., a neural network trained torecognize any of a family of parts such as standard B crimps, indentercrimps, etc. Again, the appropriate stored artificial neural network canbe loaded automatically later given an indication of the terminal orpart number to be analyzed.

Referring to FIG. 4 , the artificial neural network, e.g., trained andstored in a file as above, is used by the computer 410 or other devicehaving circuitry, e.g., an APU (Accelerated Processing Unit), indicatedat 420, which consists of a central processing unit (CPU) and a graphicsprocessing unit (GPU). Cooling airflow may be supplied by a fan 490,which may be located proximate to the APU 420, as illustrated. Computer410 includes random access memory (RAM) 430 as well as input and outputinterfaces 440, 450, respectively, for exchanging data with I/O devicessuch as remote devices 470 (e.g., USB devices, Ethernet connecteddevices, and HDMI devices, etc.). By way of example, some or all of theelements of computer 410 can be included in a single chip, a so-calledsystem-on-chip (SoC), collocated on a main board, or components includedin a computer 410 may be implemented as a distributed computing system.In one example embodiment, all of the elements of the computer 410 ofFIG. 4 , except the remote device(s) 470, are contained within a singlecase or housing having a size of about 90 mm wide, 60 mm high, and 110mm long.

The example system-on-chip (SoC) or similar circuitry outlined in FIG. 4may be implemented in a variety of devices in addition to the computer410, for example similar circuitry may be included in a cloud ornetworked device, e.g., remote device 470, or another device orplatform. In addition, circuitry other than a SoC may be utilized in oneor more embodiments. The SoC of FIG. 4 includes functional blocks, asillustrated, integrated onto a single semiconductor chip to meetspecific application requirements.

The APU 420, which may include a number of graphics processing units(GPUs), e.g., 128, and/or micro-processing units (MPUs), e.g., AMR A57Quad Core Processor, 1.43 MHz, includes an arithmetic logic unit (ALU)that performs arithmetic and logic operations, instruction decoder thatdecodes instructions and provides information to a timing and controlunit, as well as registers for temporary data storage. Again, the APU420 may comprise a single integrated circuit comprising several units,the design and arrangement of which vary according to the architecturechosen. One suitable APU is for example the JETSON NANO sold by NvidiaCorporation.

Computer 410 also includes a memory controller, e.g., comprising adirect memory access (DMA) controller to transfer data between memory430 and hardware peripherals. Memory controller includes a memorymanagement unit (MMU) that functions to handle cache control, memoryprotection, and virtual memory. Computer 410 may include controllers forcommunication using various communication protocols (e.g., I²C, USB,etc.).

Memory 430 may include a variety of memory types, volatile andnonvolatile, e.g., read only memory (ROM), RAM, electrically erasableprogrammable read only memory (EEPROM), Flash memory, and cache memory.Memory 430 may include embedded programs and downloaded software, e.g.,image processing software such as the artificial neural network programreferenced herein, etc. By way of example, and not limitation, memory430 may also include an operating system such as a Linux operatingsystem, application programs, other program modules, and program data.

A system bus 480 permits communication between various components of thecomputer 410. I/O interfaces 440/450 and network interface 460,including for example radio frequency (RF) devices, e.g., WIFI andtelecommunication radios, are included to permit computer 410 to sendand receive data to remote devices 470 using wired or wirelessmechanisms. The computer 410 may operate in a networked or distributedenvironment using logical connections to one or more other remotecomputers or databases, all of which are indicated collectively at 470.The logical connections may include a network, such local area network(LAN) or a wide area network (WAN) but may also include othernetworks/buses. For example, computer 410 may communicate data with andbetween remote devices 470.

The computer 410 may therefore execute program instructions configuredto store and analyze image data and perform other functionality of theembodiments as described herein. A user can interface with (for example,enter commands and information) the computer 410 through input devices,which may be connected to I/O interfaces 440, 450. A display or othertype of device may also be connected to the computer 410 via aninterface selected from I/O interfaces 440, 450, such as an outputinterface 450.

In one example embodiment, a system for controlling wire processingmachinery includes four main devices, i.e., a computer 410 (includingAPU 420), two cameras, and a base for mounting the cameras. In thisnon-limiting example, the APU 420 is responsible for storing theoperating system, OpenCV libraries and software for image analysis andsignal generation. An example of system specifications are as follows:ARM A57 Quad Core Processor, 1.43 MHZ or higher, Maxwell 128 Core GPU, 4GB 64-Bit LPDDR4 RAM Memory, 32 GB micro SD (image and other storage),two 3.4 mega pixel×60 FPS cameras with low light technology, gigabitEthernet port, M.2 Key E (solid state drive), HDMI Port 2.0 and/orDisplay Port 1.4, four USB 3.0 ports, and ten GPIO ports.

Referring to FIG. 5 , example components of an imaging device or system500 are illustrated. In an embodiment, two cameras 503 a, 503 b, e.g., apair of 3.4 MP USB 3.0 cameras, are included and are operatively coupledto the APU (illustrated at 420 of FIG. 4 ) to capture the upper andlower images of the object of interest, e.g., crimped wire 506 andterminal 507, weld, etc. The cameras 503 a, 503 b may be placed withinmounting structures 501 a, 501 b such that the cameras are on top ofeach other and face one another in an upper and lower configuration.Optionally, one or more LEDs, e.g., two upper LEDs 502 a, 502 b areillustrated in FIG. 5 , may be included to increase the lighting andimprove imaging quality. In the example of FIG. 5 , two upper LEDs 502a, 502 b are illustrated and two lower LEDs (not illustrated) may alsobe included in similar lower locations. The LEDs 502 a, 502 b can besecured into place using an outer shell or housing, as described inconnection with the example of FIG. 6 . In an embodiment, the cameras503 a, 503 b are separated from one another with a separation ofapproximately 30 mm to 50 mm, and in one example by about 40 mm. Bothcameras 503 a, 503 b point towards the center as illustrated in FIG. 5 .The cameras 503 a, 503 b each afford a wide-angle view, e.g., eachcovering about 120 degrees and having lenses measuring about 18 mm, asillustrated.

An object to be analyzed, e.g., a wire 506 and terminal 507, is placedin the center of the imaging device 500, e.g., manually by an operator.For manual operation, a pair of stabilizing elements 504 a, 504 boptionally may be included to assist the operator in placing the objectin view of the cameras. This orients the object to be imaged roughly inthe middle of the two cameras 503 a, 503 b. Again, this positioning canbe accomplished either manually or by some mechanical device, as theimaging analysis is not sensitive to precise alignment. In this way, thecameras 503 a, 503 b will take images of the top and bottom of theobject to have a 360-degree view of the object.

FIG. 6 illustrates an example implementation of the imaging device ofFIG. 5 , when enclosed in an outer shell. In the example view of FIG. 6, the imaging device 600 generally includes an upper part 608 a and alower part 608 b that each house a camera, one of which, i.e., camera603 b, is illustrated in the lower part in the example perspective viewof FIG. 6 . Also visible are the stabilizing devices 604 a, 604 b, whichare positioned just below the midpoint between the cameras, assisting anoperator to place the object to be identified in a correct position ontop of stabilizing devices 604 a, 604 b when manually positioning theobject of interest for imaging. In the example of FIG. 6 , the cameras,e.g., lower camera 603 b, again include lens elements that are about 18mm in diameter, and are separated by about 40 mm. The housing is about80 mm in depth, about 140 mm long, and roughly 40 mm wide, with a widerbase being optional.

An example procedure for crimp inspection in a manual press isillustrated in FIG. 7A. When an operator crimps a wire in the manualpress at 701 a, e.g., using a foot pedal that actuates a crimpingmechanism to crimp a terminal onto a wire, a cycle signal is sent to theAPU at 702 a, which receives the cycle signal via the GPIO ports, e.g.,generally indicated at 440 of FIG. 4 , or other suitable communicationmechanism. After this cycle signal is received by the APU, a foot switch(pedal control) is disabled by the APU at 703 a so that another partcannot be processed until the terminal is placed in view of the camerasfor analysis and further signal processing is accomplished, as describedbelow.

After receiving the cycle signal at 702 a, the APU opens the cameras at703 a to capture upper and lower images, as indicated at 704 a, andinitiates the recognition processing, i.e., the process of automaticallyperforming image analysis to determine if a defect is detected, asillustrated at 705 a. By way of specific example, once the operatorplaces the crimped terminal (or other object of interest to beevaluated) at the base of the imaging device, e.g., on top ofstabilizing devices 604 a, 604 b of FIG. 6 , the APU utilizes anartificial neural network, e.g., as described in connection with FIGS.2-3 , to detect the location of the terminal in the upper and lowerimages. This initial detection of the object's location using artificialintelligence allows the system to thereafter automatically locate theobject in the image, e.g., with a bounding box, even if the image is oflow quality, e.g., because the manually presented terminal and wirecombination is oriented at a random angle or is offset from the imagecenter in a random fashion due to manual placement.

Once the object has been located in the image, which may be a collectionof streamed images or video data, the system can utilize differenttechniques to make the decision at 705 a as to whether the wire andterminal combination (or other part(s) or features of interest) includesa defect, as further described in herein. If no defect is detected at705 a, the system indicates the same and processing is allowed tocontinue, as indicated to 706 a. By way of specific example, when nodefect is detected at 705 a, the system turns on a red, green, blue(RGB) LED bank, located in view of the operator, to actuate a green LEDto alert the operator that the part is good, the foot switch is enabled,and the image analysis system is put on hold for another cycle signal tostart the process again. In case the defect is detected at 705 a, theprocess is halted, as indicated at 707 a. By way of specific example, ifa defect is detected a 705 a, the system will operate the RGB LED bankto actuate the red LED to alert the operator that the part is bad andstay active to halt the system until the defect is cut (automatically ormanually) or a quality control person releases the system, e.g., byscanning an ID tag. For example, after a quality control manager hasarrived, the ID tag of the operator or the manager will be scanned, andthe system will enable the foot switch so that the cycle may startagain.

FIG. 7B illustrates an example procedure for crimp inspection inautomatic wire processing machines. In an automatic wire processingmachine, the wire or cable is cut at 701 b to prepare it for crimping.At 702 b the cable is taken via robotic or automatic process to a crimppress (if crimp terminal is to be applied) and the terminal is crimpedonto the cable. Thereafter a trigger is sent to the APU as indicated at703 b. The robotic or automatic process takes the cable to the imagingdevice cameras, as indicated at 704 b, and images are captured andanalyzed using artificial intelligence, as indicated at 705 b. If adefect is detected at 706 b, the APU outputs a NOK signal and the partis cut as defective. The process may then begin again. If the analysisdoes not find a defect at 706 b, then the OK signal is output by the APUand the normal processing is allowed to continue, as indicated at 708 b.

FIG. 8A-B illustrates an example of artificial intelligence defectdetection. As shown in FIG. 8A, a splicing element 807 a has beencrimped onto the exposed copper ends of multiple insulated wires, one ofwhich is indicated at 806 a, joining them together. The element 807 a,e.g., a crimp that splices multiple wires together, is identifiedautomatically using an artificial neural network, for example built torecognize element 807 a as crimped between multiple wire ends, joiningthem. This automatic identification permits placement of a bounding box815 a around the element 807 a in the image (here only the top or upperimage is illustrated for ease of description; however, a similarprocessing may be applied for the bottom or lower image).

The automatic identification of element 807 a removes the need forselecting a program number and using a teach-in process, commonlyrequired for conventional vision systems to locate the element 807 a andset an origin point. As such, an embodiment may identify the element's807 a location in the images reliably without being trained to theparticular set up (e.g., automatic press installation) to account forcamera angles, lighting or other variables that might perturbtraditional vision analysis systems. Further, the robust nature of theneural network identification permits the element 807 a to be reliablyidentified in a manual process without a program selection and teach-inprocess, i.e., one in which the element 807 a will be positioned forimaging manually and thus not precisely aligned.

Once the element 807 a or other object of interest has been identifiedin the image, an embodiment may place image analysis windows atpredetermined positions, e.g., for detection of given defects. Anexample of this is illustrated in FIG. 8A in which windows 809 a, 810 a,811 a, and 812 a are automatically placed around the bounding box 815 aat predetermined locations, selected for example on the basis theidentification of feature 807 a. This permits an embodiment to utilizethese areas 809 a-812 a to analyze the image for defects, e.g.,incorrect pixel color in a window selected from 809 a-812 a.

By way of specific example, FIG. 8B illustrates a defect detectedautomatically. In this non-limiting example, the defect includes lack ofinsulation or low insulation/low insertion of one wire (collectivelyindicated at 806 b). This analysis utilizes automatically detectedelement 807 b (detected using the artificial neural network, asdescribed herein), with predetermined analysis windows 809 b-811 b beingplaced on the basis of the location and orientation of bounding box 815b, programmed for the specific element 807 a and positioned according toits location detection automatically. Other factors may influence thepositioning of the windows 809 b-811 b, e.g., a scaling factor may beapplied given the size or orientation of the automatically placedbounding box 815 b. Here, the expected pixel color of area 809 b isblack or lack of copper color. However, image analysis of this area 809b (e.g., evaluation of the pixel color values) indicates a significant(e.g., over a predetermined threshold value) amount of copper colorpixels. This permits an embodiment to identify the defect, here lowinsulation/low insertion in the left side of the part, and output theappropriate NOK signal for controlling the manual crimp press orautomatic wire processing machine, as described herein.

As will be appreciated by those having ordinary skill in the art, otherobjects (e.g., different types of terminals, welds, etc.) and differenttypes of defects in addition to low insulation/low insertion may beidentified using similar processing. For example, an artificial neuralnetwork can be trained to identify any part or feature of interest. Oncethe trained artificial neural network is capable of identifying theobject of interest, suitable analysis window(s) can be automaticallyplaced (after the artificial neural network identifies the object ofinterest) for detecting defects relevant to that particular object orpart. By way of non-limiting example, other objects or features that canbe identified include splices, crimps of various type (e.g., hex,overlap, indenter, etc.), bell mouth, brush, 50/50 window, insulationcrimp wings, and ultrasonic welds. Further, defects that can beautomatically detected after identifying objects of these types includeinsulation position (as in the example of FIG. 8B, etc.), bell mouthdefects, brush defects, insulation crimp wing defects, e.g., wings bentdown or under the wire, cable seal defects, wire strands pressed on topof the conductor crimp, and the like.

In an embodiment, the object identified and the defect detected may beone and the same. That is, the artificial neural network may be trainedto identify the object defect directly. This removes the need to placeanalysis windows and perform additional image analysis, e.g., pixelcolor value evaluation. In an embodiment, both correctly formed objectsand various object defects may be directly identified by an artificialintelligence process, e.g., an artificial neural network as describedherein; this permits automatic identification of the defects and thecorrectly formed parts, as well as classification of the same forautomatic and manual press control.

As illustrated in FIG. 9 , a product may be formed via a process thatincludes an image-based inspection, as described herein and as indicatedat 901. In an embodiment, an object is visually inspected, e.g., usingartificial intelligence to automatically identify the object and/oranalyze it. The object of interest may include an insulated wire that iscombined with another element such as a terminal crimped onto the wirefor attaching the wire to another component via the terminal. After thewire or wires have passed the visual inspection, as indicated at 902,they may be included in a vehicle wire harness, as indicated at 903,which may in turn be included in a vehicle, as illustrated at 904, suchas an automobile, construction vehicle, aircraft or home appliance. Forexample, a wire harness of an automobile may include one or more wiresformed by a process such as the manual or automated crimping processesdescribed herein. Such a wire harness may be included or incorporatedinto vehicle, e.g., into a vehicle or automobile chassis, such that theprocess of making the wire harness or automobile includes theimage-based inspection(s) described herein. Therefore, an embodimentincludes a product, such as a wire harness or vehicle including a wireharness, formed via a process according to one of the embodiments.

As described herein, it will be readily understood that certainembodiments can be implemented using any of a wide variety of devices orcombinations of devices, examples of which are described in connectionwith FIG. 4 . It should be noted that the various functions describedherein may be implemented using instructions stored on a memory, e.g.,memory 430, that are transmitted to and executed by a processor, e.g.,APU 420. Computer 410 includes one or more storage devices that storeprograms and other data. A storage device, as used herein, is anon-transitory storage medium. Some additional examples of anon-transitory storage device or medium include, but are not limited to,storage integral to computer 410, such as a memory or a solid-statedrive, and removable storage, such as an SD card or a memory stick.

Program code stored in a memory or storage device may be transmittedusing any appropriate transmission medium, including but not limited towireless, wireline, optical fiber cable, RF, or any suitable combinationof the foregoing.

Program code for carrying out operations may be written in anycombination of one or more programming languages. The program code mayexecute entirely on a single device, partly on a single device, as astand-alone software package, partly on a device and partly on anotherdevice, or entirely on another device. In an embodiment, program codemay be stored in a non-transitory medium and executed by a processor toimplement functions or acts specified herein. In some cases, the devicesreferenced herein may be connected through any type of connection ornetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made through other devices (for example,through the Internet using an Internet Service Provider), throughwireless connections or through a hard wire connection, such as over aUSB connection.

Example embodiments are described herein with reference to the figures,which illustrate example methods, devices and program products accordingto various example embodiments. It will be understood that the actionsand functionality may be implemented at least in part by programinstructions. These program instructions (computer code) may be providedto a processor of a device to produce a special purpose machine, suchthat the instructions, which execute via a processor of the deviceimplement the functions/acts specified.

It is worth noting that while specific elements are used in the figures,and a particular ordering of elements has been illustrated, these arenon-limiting examples. In certain contexts, two or more elements may becombined, an element may be split into two or more elements, or certainelements may be re-ordered or re-organized as appropriate, as theexplicit illustrated examples are used only for descriptive purposes andare not to be construed as limiting.

As used herein, the singular “a” and “an” may be construed as includingthe plural “one or more” unless clearly indicated otherwise.

This disclosure has been presented for purposes of illustration anddescription but is not intended to be exhaustive or limiting. Manymodifications and variations will be apparent to those of ordinary skillin the art. The example embodiments were chosen and described in orderto explain principles and practical application, and to enable others ofordinary skill in the art to understand the disclosure for variousembodiments with various modifications as are suited to the particularuse contemplated.

Thus, although illustrative example embodiments have been describedherein with reference to the accompanying figures, it is to beunderstood that this description is not limiting and that various otherchanges and modifications may be affected therein by one skilled in theart without departing from the scope or spirit of the disclosure.

What is claimed is:
 1. A method of operating wire harness manufacturingequipment, comprising: adding, using the wire harness manufacturingequipment, an element to a wire to form a combination of the element andthe wire; capturing, using an imaging device, one or more of an upperimage and a lower image of the combination; analyzing, using one or moreprocessors operatively coupled to the imaging device, the one or more ofthe upper image and the lower image to detect a defect; the analyzingcomprising using a model pre-trained with a random set of trainingimages including one or more objects of interest disposed therein; andthereafter indicating that the defect has been detected.
 2. The methodof claim 1, wherein the analyzing does not utilize data supplied duringa teach-in mode for the wire harness manufacturing equipment.
 3. Themethod of claim 1, wherein the analyzing comprises identifying theelement as the object of interest in one or more of the upper image andthe lower image and thereafter identifying the defect in one or more ofthe upper image and the lower image.
 4. The method of claim 3, whereinthe analyzing comprises, after identifying the element, examining one ormore predetermined regions in one or more of the upper image and thelower image.
 5. The method of claim 4, wherein the one or morepredetermined regions are automatically identified in the upper imageand the lower image using location information automatically obtainedfor the element.
 6. The method of claim 1, wherein the capturingcomprises moving the combination manually.
 7. The method of claim 1,wherein the analyzing comprises utilizing an artificial intelligenceprogram to detect the defect.
 8. The method of claim 7, wherein theartificial intelligence program comprises a trained neural network. 9.The method of claim 8, wherein the trained neural network has beentrained to identify the element.
 10. The method of claim 9, wherein thetrained neural network has not been trained to identify the defect. 11.The method of claim 1, wherein the element is one or more of a crimp, aseal, and another wire.
 12. The method of claim 1, wherein the defect isone or more of insulation position defects, bell mouth defects, brushdefects, cut off tab defects, crimp wing defects, incorrect wing bendingdefects, cable seal defects, and wire strands pressed on top of aconductor crimp defects.
 13. A method, comprising: adding, using wireharness manufacturing equipment, an element to a wire to form acombination of the element and the wire; capturing, using an imagingdevice, one or more of an upper image and a lower image of thecombination; analyzing, using one or more processors operatively coupledto the imaging device, the one or more of the upper image and the lowerimage to detect if a defect is present; the analyzing comprising using amodel pre-trained using a random set of training images including one ormore objects of interest disposed therein; thereafter indicating thatthe defect has not been detected; and incorporating at least thecombination of the element and the wire into a wire harness.
 14. Themethod of claim 13, wherein the wire harness is one of a vehicle wireharness and an appliance wire harness.
 15. The method of claim 14,comprising one or more of: incorporating the vehicle wire harness into avehicle; and incorporating the appliance wire harness into a homeappliance.
 16. The method of claim 15, wherein the vehicle is one ormore of an automobile, a construction vehicle, and an aircraft.
 17. Amethod, comprising: obtaining, from an imaging device, one or more of anupper image and a lower image of an element added to a wire to form acombination of the element and the wire; analyzing, using one or moreprocessors, the one or more of the upper image and the lower image todetect a defect; the analyzing comprising using a model pre-trainedusing a random set of training images including one or more objects ofinterest disposed therein; and thereafter indicating that the defect hasbeen detected.
 18. The method of claim 17, wherein the obtainingcomprises moving the combination manually to the imaging device.
 19. Themethod of claim 17, wherein the analyzing comprises using an artificialneural network to identify the element.
 20. The method of claim 17,wherein the analyzing comprises using an artificial neural network toidentify the defect directly.